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ABSTRACT 



Internal Participatory Evaluation as an Organizational Learning System: A Longitudinal 

Case Study 

Tim Robinson Ph D. & Brad Cousins Ph D. 



Basic Message of Presentation 

Over the past few decades our knowledge about how and why evaluation data are used has 
expanded considerably. This knowledge prompted the development of a practical form of 
collaborative inquiry called '"participatory evaluation/’ While participatory evaluation has been 
demonstrated to enhance the utilization of specific evaluation results, it also has great potential for 
stimulating the development of organizational learning capacity. However, organizations have 
been slow to embrace methods of systematic inquiry such as program evaluation and only a small 
number of studies have looked at the effects of evaluation on the capacity of organizations to learn. 

It would seem that internal evaluators, given their position within organizations and 
expertise with the entity being evaluated, would be ideally suited to conducting practical 
participatory evaluation. But the research looking at the participatory approach as an 
organizational intervention has emerged predominantly from situations where the evaluator is 
external to the program being evaluated. Few studies exist that show the impact of practical 
participatory evaluation when the evaluator is not only a member of the organization, but also has 
program knowledge and expertise. 

This paper addresses this gap in the literature by reporting on a two and one-half year 
longitudinal single case study of practical participatory evaluation of a national, publicly funded 
training organization. In particular, the paper examines one of the key dimensions of 
organizational learning, levels of learning. Key questions addressed include: 

• Can practical participatory forms of evaluation cause organization members to question 
basic organizational assumptions? 

• Does involvement in participatory evaluation impact the organization beyond the scope of 
the evaluation project itself? 

Drawing on multiple sources of qualitative data, the authors show that quite significant 
organizational effects can result from such an intervention. These findings add value to prior 
research in this area that has revealed only low-level or “single loop” learning effects. 
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Implication to Practice or Theory 

An important feature of the current study is the distinction that was made regarding which 
type of internal evaluator the present study’s author fulfilled. A conventional definition of an 
internal evaluator is someone employed by the organization and who conducts evaluation with 
program personnel. The present study investigated the effects of an internal evaluator who was 
both employed by the organization and internal to the program being evaluated. The study’s 
findings do offer some insights regarding this issue. For example, the study shows that the added 
dimensions of evaluator knowledge of both the program and organizational context can have a 
powerful impact on developing the organization’s capacity to inquire systematically and to learn. 

The paper also contributes to the organizational learning research knowledge base by 
demonstrating the differential effects on various sub-processes and strategies associated with the 
organizational learning construct. Finally, the findings provide researchers and practitioners with 
useful knowledge about the circumstances under which internal participatory evaluation is likely to 
be effective. 



Presenter: Tim Robinson , Ph D. is the Manager. Sales Force Career Development, Canada Post 
Corporation 

Discussant. Brad Cousins , Ph.D. is the Director of Professional Development & Professor, 
Faculty of Education, University of Ottawa 
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Internal Participatory Evaluation as an Organizational 
Learning System: A Longitudinal Case Study 

Contemporary evaluators and evaluation scholars have come to understand the power and 
potential of alternative, particularly collaborative forms of evaluation, in enhancing the utility of 
program evaluation. Recent surveys of evaluators on such matters (Cousins, Donohue & Bloom, 1996; 
Preskill & Caracelli, 1997) have confirmed that this group perceives quite significant impact on 
program practitioners and the organizations within which they work of evaluation approaches that 
directly embrace the involvement of evaluation stakeholders. Such impact may be directly 
attributable to the findings of the evaluation in terms of support for discrete program decision making 
(instrumental use) or enhanced understanding and learning about program processes and effects 
(conceptual use). When stakeholder involvement in collaborative evaluation is substantial and long- 
lasting, quite enormous effects can also be attributed to evaluation implementation. This is what 
Patton (1997) termed "process use" or "individual changes in thinking and behavior, and program or 
organizational changes in procedures and culture, that occur among those involved in evaluation as a 
result of the learning that occurs during the evaluation." (p. 90). 

Process use at the individual level is manifest in the acquisition and development among 
program practitioners and participants of knowledge and skills associated with systematic inquiry and 
evaluation logic. These abilities, it can be argued, may then be utilized in other aspects of the roles of 
program designers and implementers. Regardless, they are likely to extend beyond the particular 
program that serves as the focus for evaluation (Shulha & Cousins, 1997). Evaluation effects 
independent of the findings of evaluation can also be observed at the organizational level. Recently, 
many evaluators and evaluation theorists have written about the role that evaluation plays in fostering 
organizational learning (Cousins, 1998; Cousins & Earl, 1992, 1995; Jenlink, 1994; Owen & Lambert, 
1995; Preskill & Torres, 1998; Torres, Preskill & Piotnek, 1996). Although Preskill and Caracelli 
(1997) differentiate the terms, organizational learning may be conceived to be an instance of process 
use at the organizational, as opposed to individual, level. 

Despite the recent practical and scholarly interest in the relationship between evaluation and 
organizational learning, research-based knowledge in support of this connection is limited. Forss, 
Cracknell and Samset (1994) conducted a retrospective study of several European evaluations and 
found only limited, incremental or “single-loop” organizational effects. They question evaluation's 
potential to precipitate deeper, more penetrating or “double-loop” organizational changes such as how 
organizational members view the world and take collective action. Studies in the collection edited by 
Cousins and Earl (1995) examined the effects of participatory evaluation on utilization and 
organizational learning. This collection of retrospective case studies provided very limited evidence 
of double-loop organizational learning. The editors concluded that evaluation-prompted changes to 
organizational culture are only likely to be in evidence after sustained engagement of organizational 
personnel in evaluation activities. Cousins (1999) differentiated organizational process use effects at 
various levels within a school district. He investigated effects at the level of the research team, the 
secondary school in which participatory project was implemented, the district central board office and 
other schools within the district. Although some effects were observable. The author also concluded 
that sustained evaluation activity within the district would be required in order to stimulate substantial 
changes in organizational culture. 

Despite the promise that collaborative forms of evaluation can act as an organizational 
learning system to facilitate the transfer and integration of knowledge and “learning” from the 
individual to the organization, evidence supporting this claim remains thin. This may be the case for 
two reasons. First, extant empirical investigations in the domain tend to be retrospective case studies 
or wider quantitative surveys of evaluation practice. This, in spite of our longheld understanding that 
longitudinal research designs are likely to be the most productive means of studying evaluation 
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utilization (Cousins & Leithwood, 1986). Second, the psycho-social construct 'organizational learning' 
is very complex, ill-defined, and thus difficult to measure (Argyris, 1993). Given our current limited 
understanding of the construct, a qualitative case study approach would be the preferred mode of 
inquiry. Thus, the purpose of the present study is to examine the relationship between evaluation and 
organizational learning having taken both of these deficiencies into account. The investigation is a 
single case study of a national Canadian training organization that implemented a comprehensive 
internal participatory evaluation of its programs over a protracted period of time. The study was 
guided by two basic questions. (1) What are the consequences of internal participatory evaluation on 
organizational learning capacity? (2) What factors explain the extent to which these consequences 
occurred? 



CONCEPTUAL FRAMEWORK 

The conceptual framework guiding the study appears in diagrammatic form in Figure 1. The 
first consideration is the context in or organizational frame within which learning capacity is 
embedded. Anderson's (1982) framework for organizational climate serves as a useful heuristic 
perspective of organizational factors likely to influence organizational learning capacity. Her 
framework divides climate into four subcomponents: (1) ecology, or human relationships to the 
physical and material aspects of the organization; (2) milieu, a psycho-social dimension concerned 
with the presence of persons or groups and their characteristics; (3) social system, or patterned 
relationships among persons and groups; and (4) culture, the belief systems, values, cognitive 
structures and mental models shared among organization members. A second source of influence, also 
represented in Figure 1, is the conditions and factors associated with the environment within which the 
organization exists. Environmental influences not only play a role in shaping the organizational frame, 
but they are affected by organizational actions. To that extent, the conceptual framework is non- 
recursive since the flow of influence is not unidirectional. 

Organizational learning capacity is the dependent variable in the present study. The term 
“capacity” reflects both the dimensions of organizational learning, as well as the processes that act to 
facilitate its growth. In the present study three dimensions identified in the conceptual framework 
emerged as effects that were particularly sensitive to evaluation as an intervention. These were shared 
knowledge representation, organizational memory, and levels of learning. These three dimensions and 
the factors and/or processes that influenced their development are described in turn. 

Shared knowledge representation . Although many approaches to the study of organizational 
learning have been based on theories of individual learning, organizational learning is more complex. 
As Fiol and Lyles (1985) states, “Though individual learning is important to organizational learning, 
organizational learning is not simply the sum of each member’s learning” (p. 804). While the term 
“learning” remains essentially the same as in the individual case (i.e., to increase one’s capacity to take 
effective action), the learning process increases in complexity and challenge as the learning involves a 
much larger group of diverse individuals (Kim, 1993). As such, shared knowledge representation in 
organizations acknowledges that understanding how individuals learn is at the core of understanding 
how organizations learn. Typically, decision-making processes and individual perspectives are stored 
collectively by organization members as images and maps or theories-of-action (Argyris & Schon, 
1978), routines (Levitt & March, 1988), mental models (Senge, 1990), meaning structures (Dixon, 
1994), or assumptions (Drucker, 1994). Mental models, for example, control what an individual pays 
attention to, how she interprets information and concrete experiences, and chooses to act. Thus, when 
different mental models or knowledge representations are scattered among organization members, 
"organizational learning disabilities" are likely to occur such as the inability to reach consensus on a 
particular strategy and take collective action (Senge, 1990). Conversely, when mental models are 
widely held or shared by organization members, the capacity for organizational learning is significant. 



Levels of learning . Most theorists would agree that organizations vary in terms of the level of 
learning they experience. Learning can range from incremental, low-level or "single-loop" learning to 
high-level or "double-loop" learning (Argyris & Schon, 1978; Fiol & Lyles, 1985; Huber, 1991). Fiol 
and Lyles (1985) defined low-level learning as a phenomenon occurring within the existing 
organizational structure and its given sets of rules. Indicators or evidence that single-loop learning has 
occurred could include, for example, the minor adjustments and fine tuning of existing organizational 
decision-making processes. In this respect it is incremental. High-level or double-loop learning, on 
the other hand, is reflected in the alteration of the overall decision rules, norms, and beliefs of the 
organization. It occurs when individual mental models integrate with mental models held by other 
organization members to generate new collective theories of action. This integration depends on 
organization members exposing and challenging individual and organizational assumptions that are 
typically not available for public discourse (Argyris, 1993; Kim, 1993). Evidence to support the 
existence of double-loop learning could include organization members altering existing assumptions or 
perspectives that guide the organization and its programs. Thus, for double-loop learning to occur new 
collective perspectives or mental models must first be created, then codified and stored in the 
organization’s memory. This, in turn, will result in long term impact on the organization in terms of 
future decision making and action (Simon, 1991). 

Organizational memory. An important dimension of organizational learning is reflected in the 
organization’s ability to organize, store and retrieve information (Levitt & March, 1988; Simon, 1991). 
Levitt and March (1988) described memory in terms of the various routines and frameworks which are 
developed by the organization to guide its behavior. Routines include the forms, rules, operating 
procedures, and technologies that drive the organization, whereas, the organizational frameworks 
contain the structure of beliefs, shared mental models, and culture that allow the organization to 
function efficiently day-to-day (e.g., survive turnover of key personnel). An organization’s memory is 
critical to organizational learning for it houses the shared experiences and mental models valued by the 
collective for use in future situations. For example, they enable the organization to self-correct in 
response to environmental change or to transform itself in anticipation of a desired future (Dixon, 
1994). 

The processes that act to foster or inhibit organizational learning in organizations (e.g., action 
learning, knowledge generation and acquisitions, and interpretive systems) have received a good deal 
of attention in the literature (Drucker, 1992; Huber, 1991; Levitt & March, 1988). For the purpose of 
this paper, the central question to be addressed is how do these processes affect the establishment of 
shared knowledge and mental models, as well as their codification to the organization’s memory? The 
assumption that knowledge is socially constructed (Bandura, 1986) helps to frame an answer. As 
discussed earlier, although members of the organization can learn individually, organizational learning 
occurs when these individual understandings are shared among organization members and memories 
are altered or acquired (Fiol & Lyles, 1985). 

The learning system described in Figure 1 is internal, practical participatory evaluation. This 
learning system is generative rather than adaptive (Cousins, 1996; Huber, 1991; Senge, 1990) because 
it’s primary function is to create new knowledge rather than acquire and adapt knowledge from the 
organization's environment. The likelihood that this evaluative approach will serve as an 
organizational learning system and assist with the transfer and integration of individual learning to the 
organization, is augmented by virtue of the collaborative processes at play - such integration occurs as 
a result of an intense participatory process. By participatory is meant, a trained evaluator working in 
partnership with members of the program community over an extended period of time to do an 
evaluation. In practical participatory evaluation (Cousins & Earl, 1995; Cousins & Whitmore, 1998) 
the evaluator normally brings to the partnership technical knowledge and expertise about systematic 
inquiry and evaluation logic. Program practitioners and other participants, on the other hand, bring 
knowledge of program logic and organizational culture and processes. In the present instance, the 
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evaluator was also internal to the organization within which the evaluation occurred and to the 
program which served as the object of evaluation. In this respect, the trained evaluator also had 
programmatic and organizational knowledge and expertise. Practical participatory evaluation 
generally locates as follows on three primary dimensions (Cousins & Whitmore, 1998). First, 
technical decision making about the evaluation is shared between the evaluator and the program 
practitioners. Second, program practitioners not trained in evaluation usually participate quite 
extensively in a full range of evaluation activities, and finally, participation is normally limited to 
primary users or program practitioners who are in a position to do something with the knowledge 
generated by the evaluation. 

As mentioned above, two types of impact are to be expected from collaborative evaluation. 
First, such forms of inquiry are likely to enhance conceptual and instrumental consequences of 
evaluation by virtue of the process engendering a strong sense of ownership and deep understanding of 
observed findings. This we term use of evaluation findings. Second, and what will be the focus of this 
paper, participatory evaluation, by virtue of program practitioners’ active involvement in the 
implementation of the evaluation is likely to enhance process use. This we represent as organizational 
learning: it is understood that such learning (i.e., double-loop) will result if individual learning and 
perspectives are effectively integrated into the organization. 

METHOD 

A longitudinal single case study design was used for the present exploratory research. This approach 
provided the flexibility to explore the functions and impact of internal participatory evaluation within a 
guiding framework tied to organizational learning. Qualitative methods designed to capitalize on the 
prior specifications of a conceptual framework were therefore selected for use. 

Case Organization and Focus of Evaluation 

The program selected for the study was an evaluation of a national leadership training program 
that is directed, funded, and implemented by a partnership of Canadian federal and provincial 
governments and national organizations in Canada. The first author was employed by the case 
organization (CNT) and had, as an explicit and significant part of his role, responsibility for the 
evaluation. Over a two-year period, the first author (fulfilling the role as the study’s researcher) 
worked in partnership with members of the project’s steering committee to design and implement a 
comprehensive evaluation plan which included a detailed job task analysis, needs assessment, and 
quantitative and qualitative data collection. 

Data Collection 

Retrospective reflections. The researcher systematically reviewed existing documents (e.g., 
minutes of meetings, position papers) and recorded on audiotape his reflections of previous discussions 
that pertained to the central issues of the present study. This was an important step for it established 
the evolutionary process which both the program and the organization had undergone prior to the 
initiation of the evaluation project. 

Participant observation. Data were collected via participant observations during the evaluation 
as interesting or important events occurred. The observations were guided to some degree by the 
following questions: (1) what effect is the researcher having on the participatory process? (2) in what 
ways is the internal participatory process evolving? (3) what organizational impact is the internal 
participatory process having? and (4) what factors explain the observed impact? The observation notes 
were somewhat unstructured and recorded onto an audio-tape. Participant observations were collected 
up to and including the end of the evaluation project (April, 1997), that is, once evaluation data had 
been analyzed, recommendations formulated, and a final report prepared and disseminated throughout 
the country. 
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Interviews. During the evaluation project, 27 structured interviews were carried out with 
members of the evaluation project’s steering committee (i.e., three interviews per committee member). 
One other interview was carried out with the organization’s president to bring the total interviews to 
28. Figure 2 provides an overview of the committee’s membership. 

An independent interviewer was used in order to obtain corroborating evidence while at the 
same time minimizing participant observer bias. As outlined earlier, the first author was employed by 
the case organization and had as an explicit part of his job the responsibility to facilitate the evaluation 
project. Given that the author simultaneously acted as both evaluator and academic researcher, the 
potential for bias needed to be countered. Thus, the present study employed an external, independent 
interviewer to implement this data collection method. The audio tapes were transcribed by an external 
support person as well. 

It is important to note that neither the tapes nor the transcripts that resulted from the interviews 
were made available to the researcher until after the final rounds of interviews (i.e., following the 
completion of the evaluation project and release of its final report). 

Archival data. The researcher collected records and documents that addressed specifically or 
provided a context for the phenomenon being observed. Archival evidence was collected from 
historical documents, as well as ongoing sources of information (e.g., minutes of meetings). 

Focus group. A focus group interview was conducted with the respondents interviewed 
throughout the project approximately six months following the completion of the evaluation project. 
The focus group was led by the second author so as to minimize the intrusion of familiarity bias given 
the committee’s working relationship to the first author. The primary purpose of the focus group was 
to collect information that represented the group’s (i.e., Planning and Evaluation Committee) 
perspective regarding the impact that the internal participatory evaluation process had on the 
organization. It also provided an opportunity for the respondents to provide feedback on the findings 
of the study thereby adding to their credibility. A time period of approximately six months was 
judged by the researchers to be sufficient to allow for impact of the evaluation, both its findings and its 
process, to become evident. 

Plan of Analysis 

Coding of data. Procedures for coding these data were based on those described by Miles and 
Huberman (1984, 1994) and a list of “start codes” grounded in the conceptual framework specified 
above was applied. The coding process was as follows: 1) the transcripts and archival data were read 
and divided into meaning units (i.e., part sentences, sentences, multiple sentences, or paragraphs that 
address a single theme or issue); 2) the data were reviewed a second time as they were entered into the 
database; and 3) each meaning unit was subsequently read for a third time as the codes were applied. 
This iterative process allowed the original list of start codes to be refined and new codes to be added. 

Inter-coder reliability . Three dimensions of agreement between the researcher and an 
independent analyst were analyzed: 1) agreement about first order codes; 2) agreement about second 
order codes; and 3) agreement about causal inferences. Agreement was defined as both the researcher 
and the independent analyst selecting the same code. The first author and the independent analyst were 
generally consistent in identifying first order codes and causal inferences. A drop in consistency in 
agreement about second order codes was noted and may be partially due to the greater variability in the 
number of codes possible for selection. 

Computerization of Analyses. Data from all sources were subsequently sorted and 
categorized using a database management system called “Visual FoxPro 5.0 for Windows.” A template 
specifically designed for this study was developed and provides the necessary tools to manage data, 
whether organizing tables of information and running queries, creating an integrated relational 
database management system, or programming a fully developed data management application for end 
users. 
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FINDINGS 






Prior to a presentation of the study’s findings, a general description of the case organization 
will assist the reader put the findings in context. 

Subsequent to the launch of the evaluation intervention, CNT members with specific program 
responsibility were implicitly encouraged to work independently to solve problems that affected the 
programs directly under the individual’s control. Senior management’s view was that by promoting 
this kind of individualism and ownership an increased sense of responsibility would be fostered. This 
in turn, it was believed, would lead to increased productivity 

Not surprisingly, when it came time to make decisions on program initiatives or organizational 
policies, members often acted alone or in concert with only the senior-level staff. CNT members were 
rarely called upon to be apart of the collective decision-making processes involving program areas or 
issues outside their jurisdiction. Boundaries were clearly defined around the various programs, and 
projects offered by the organization and members became quite comfortable in managing their stated 
portfolios. In addition to affecting the organization’s internal operating procedures, this style of 
conducting business extended to CNT’s external relationships. Dealings with partner organizations 
within the system were relegated to somewhat formal settings (e.g., semi-annual meetings) that dealt 
with policy issues aimed at maintaining the program’s status within the system. These interactions 
provided little opportunity for program leaders to exchange ideas or beliefs central to the design or 
delivery of the program and, consequently, little change in the program had been observed since its 
inception 20 years ago. Furthermore, organizations within the system established rigid jurisdictional 
control over specific program areas and became quite defensive of their “slice of the program pie.” 

Given this environment and the limited time members of either CNT or partner organizations 
engaged in meaningful discussions, the organization was involved in relatively low level or single-loop 
learning. Such limits on organizational sense making were reflected in the nature of the changes that 
had occurred to the program during the previous 1 5 years, the degree of collective thinking and 
decision making taking place, and the defensive mechanisms that had developed to protect the 
organization. 

The findings associated with the three dimensions - shared knowledge representation, levels of 
learning, memory — are presented briefly below. An elaborate treatment of the effects on each of these 
variables is beyond the scope of this paper. Table 1 summarizes how the evaluation process unfolded 
during the study, as well as the general purpose of each time period (TP). Each section therefore 
begins with a graphical representation of the extent to which each variable changed over the course of 
the evaluation project. The trends are presented as a line graph beginning at the baseline (TP 1) or 
before the project was initiated up to the project’s report being released (TP 4). 
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Prior to the start of the project, members from partner organizations were distrustful of CNT 
and this limited individuals from sharing their views about important program issues. However, once 
the project was initiated, the collective understanding of the issues improved as members became 
implicated in the day-to-day operation of the project. As one member remarked, “There was a very 
clear mandate given to committee members to raise the issue of program revision with not only their 
constituents but with any other constituents that involve program participants.” While the expressed 
purpose of these discussions was to inform stakeholders about the evaluation project and their role in 
it, the evidence seemed to suggest that these sessions assisted in confirming viewpoints about program 
issues that have been batted about for years. By the end of the second time period, although there were 
still some members harboring feelings that important, fundamental issues needed to “get out on-the- 
table”, a climate for sharing and learning from one another was being established. 

A notable improvement was reported by respondents in this regard as the evaluation data 
began to emerge and committee members began to take an active role in data analysis. A significant 
consolidation and convergence of views around such issues as: participants’ needs and conceptual 
design resulted from an increased dialogue among members during this intense work phase. As one 
member representing the national organizations reported: 

I think I have a greater understanding of the issues from actually working with [the researcher] 
and others in the group over the last eleven months. I’m recording discussion and I’m doing 
the national organization’s report. (TP 3, steering committee member, Megan) 

As the project’s report was being prepared for release, increased clarity and like-mindedness 
among committee members was observed. “ We have been able to bring the national organizations’ 
position to a point that there are four or five key themes that are starting to emerge” (observation field 
notes, April 30, 1996). 

Two factors appeared to be predominantly responsible for the enhanced states of shared 
knowledge acquired by organizational members - the participatory evaluation process and the impact 
of the researcher. First, the participatory process stimulated enhanced commitment to the evaluation 
exercise and this led to members taking responsibility for various aspects of the evaluative process 
such as facilitating stakeholder forums. 



Information within square brackets in all quotations represents 1) substitution for a participant’s name to protect 
confidentiality, 2) substitution for pronoun not traceable in quoted segment, or 3) implied statement by participant 
based on information immediately preceding in the interview or participant observation. The date and source 
(time period, role) appear in parenthesis. Fictitious names for participants are used. 
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Direct involvement in what can be described as ‘learning forums’ increased committee 
members’ understanding of both the process, and more importantly, the underlying program and 
associated organizational issues. A member of the steering committee revealed his perception of the 
committee’s role: 

I think the primary role of the [steering committee] is to make sure that all the partners are 
really involved in [the project]: they're informed, they get a chance to provide input, they get a 
chance to provide feedback, to react to what we collect, and then discuss and analyze it in 
terms of impact on their program and system. (TP 2, steering committee member, Henri) 

A more open attitude resulted and this increased the project leaders’ sensitivity to the importance of 
providing key stakeholder groups with opportunities to critique the work of the committee. One 
member commented on the importance and benefit of providing these forums for sustained 
interactivity: 

I think [the meeting] was a really important one because there were a lot of people at the table, 
many of whom weren’t really all that up to speed with what we were hoping to accomplish 
with the program evaluation. A lot of ideas were put on the table and there was a lot of 
challenge to the people that had designed the initial plan... that made us really question some 
of the assumptions that they were making. (TP 2, steering committee member, Megan) 

This work not only improved their understanding of the issues but gave them confidence in 
their views. It is also important to note that these individual constructions and mental models would 
not have been successfully shared without the number of forums for sustained interactivity that kept 
the issues “top-of-mind” for members. 

On the other hand, there were reports that significant program and organizational issues were 
still “ lurking in the background somewhere” and not being dealt with. As one member commented: 

...the dialogues are happening in the corridors, in the offices, in the regions... independently 
though. I believe independently. (TP 2, steering committee member, Charles) 

Fortunately, as the project reached its latter stages, very few issues were left unearthed. Once 
again, the ongoing intensive nature of the group’s interventions was highlighted as being particularly 
important in building near unanimous consensus on a large number of diverse program and 
organizational issues. The actual work by members to create the data collection tools, implement the 
various data collection steps, analyze the data, and write the report all contributed greatly to the 
eventual development and acceptance of shared mental models that resulted. Interestingly, one 
respondent made implicit mention of this phenomena by comparing and contrasting his personal 
relationship with another member of the committee over the course of the project: 

At one time [committee member] and myself were really arguing on every single issue... 
fighting all the time. But now, because of our work together, we’re together. . . trying to figure 
out a way to approach this or that and to make sure that the ideas we share will carry. Now I 
feel that we’re pretty good accomplices and thinking on behalf of the program not just 
ourselves. (TP 4, steering committee member, Wes) 

The researcher made important contributions throughout the project to the creation of a climate 
conducive to the unearthing of personal beliefs among members and generally holding together the 
project team as it struggled through some difficult and stormy phases. Not surprisingly, the 
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researcher’s group facilitation skills were mentioned by several respondents as having an important 
effect on the climate established for members to engage in sessions where personal beliefs and existing 
mental models would surfaced and then be tested. For example, respondents made mention of how the 
researcher first “ slowed down the process to include everybody in building group consensus.” One 
member commented that the researcher’s approach was intentional in the hopes that information and 
individual perceptions would be shared and group interaction would be fostered: 

I'm not sure what type of leadership [the researcher] would have... participatory certainly. He's 
very, very careful about getting people informed, involved; it's a very big concern of his to 
make sure that the partners get the required information, are kept informed of what is going on, 
and given a chance to think and react to the information. (TP 2, steering committee member, 
Henri) 

Once comfortable, members were then able to share their personal beliefs and conceptions that up until 
the evaluation project were not discussible, and more importantly, not tested. One member elaborated 
on a specific intervention technique he found particularly effective: 

...[the researcher] challenges people to be open and involved. Just the way he set up the 
agenda last time, the very first item was "let’s go around the table and let’s talk about how 
everyone feels about the whole process." I found that really useful for me because it allowed 
me to express my feelings of being kind of overwhelmed by the whole thing, ... it allowed 
everybody to do that, to really talk about what’s important and to challenge others. I mean, he 
really encourages that to happen so people are not allowed to be passive. (TP 3, steering 
committee member, Brad) 

The researcher’s academic background, research skills, and program expertise were also 
identified by respondents as being directly linked to process use at the individual level: 

It’s been a learning experience for everybody that’s for sure. Obviously, the way [the 
researcher] has been able to lead this project is directly related to the success of the study. If 
[the researcher] didn’t have the academic background in program evaluation I think we would 
have a much harder time. For me at least, it’s been a really valuable learning experience to 
understand some of these differences in terms of research, the use of data and evaluations. (TP 
3, steering committee member, Steve) 

Group learning was facilitated by the researcher’s capacity to helped members clarify the relationships 
in the data, see links with other members’ views, and connect past events with current thinking. In 
addition, his 15 years of experience in the program was felt to be especially useful in sorting and 
filtering the myriad of information generated by the project, as well as accessing ‘micro-political’ 
information that was necessary for a deep understanding of the issues. Finally, for a few members, the 
researcher’s technical training as an evaluator contributed to 1) a general sense of confidence that they 
were on the right track and 2) their personal development regarding knowledge and skills associated 
with program evaluation. 
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The ability of the organization to collect, organize, store, and utilize new conceptual 
frameworks or mental models for decision-making showed improvement as the evaluation project 
evolved. This dimension was rated as “ moderate” at the end of the project as only a limited amount of 
examples were observed. 

Two concrete actions were undertaken by the organization and its members to assist with 
storing and distributing information and/or shared mental models. The first was the creation of a 
simple information storage and retrieval system - a set of binders that were temporally organized and 
contained all project information, including all new ideas, thoughts, and issues under consideration. 

The binders were continually updated and distributed to both committee members and key stakeholders 
groups within the system. Previously, information of this nature was never recorded in a manner which 
could be easily referenced or shared; it was typically held by individual organizational members 
responsible for a specific program, or embedded in the individual memories of senior management 
staff. This was one of the few times, sensitive program and organizational issues were exposed for 
public scrutiny and discussion. One member describes its effects: 

The project leader has created a binder of information that he’s made sure every partner gets 
and he gives them updates on a regular basis. I think that’s clearly what the partners need, 
because it creates a sense of trust, it creates a sense of understanding... because the issues are 
tracked. (TP 2, steering committee member, Steve) 

The second initiative involved the purchase and extensive use of an internet-based electronic 
mail system. This was viewed as an important ‘system’ modifier that assisted members to 
communicate their thoughts, feelings, and individual mental constructions to colleagues as they were 
being formed across the country. Keeping people talking, sharing ideas, challenging each other outside 
of formal meetings, was critical to keeping individual mental models explicit in the hopes that shared 
models would eventually result. Particularly noteworthy conversations were regularly downloaded and 
distributed (hard copy) to members prior to formal committee meetings to highlight important 
contributions to a specific issue or conception. Consequently, the email system, in conjunction with 
the binders, emerged as a sort of repository of the committee’s views - past and present. 
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Evidence that organizational double- loop learning occurred ranged from discussions which 
probed the underlying personal assumptions about the purpose of the program to the traditional roles 
and decision-making processes used by the organization. 

The range of program issues debated by various stakeholder groups provides a gage of the 
extent of organizational learning encountered. Perhaps the most significant individual learning event 
resulted from members becoming immersed in an ongoing discussion aimed at uncovering the 
program’s validity assumptions, that is, beliefs about how the program is supposed to work. At a 
committee meeting in January 1996 and an organization meeting in February 1996, the underlying 
beliefs as to how the program, and its various steps, was supposed to cause improvements to 
participants’ behaviors was freely discussed and challenged. “There isn’t anything articulated 
anywhere in terms of how the program is to operate. While we have three components with which 
everyone is familiar, there has never been anything explicitly written down. One of the real treasures 
of the evaluation will be to have the committee state a program theory in terms of cause-effect 
relationships and this should provide some very useful guidelines to program designers in the future as 
they go about reconstructing this thing” (observation field notes, February 22, 1996). 

In terms of learning at the organizational level, the evaluation exercise was clearly having 
impact on the organization’s standard operating procedures used to respond to challenges from partner 
organizations. As one member commented: 

[The project] is the first true example of full involvement [by the partners] in decision making. 
Historically, decisions about [the program] have pretty much been driven by [the 
organization]. Now, people expect to be involved. This is something that we had to learn ... 
we are not the only people being impacted by these decisions. Historically, we were pretty 
directive when dealing with partner concerns and we have found out that no matter how 
forceful you want to be, if people don’t feel that your decision makes sense or that the decision 
is not theirs... you can be as forceful as you want but it’s not going to work. (TP 3, steering 
committee member, Henri) 

By the end of two and half years of ongoing discussion, debate, and consensus-building, there did seem 
to be evidence emerging that individual mental models were becoming incorporated into a single or 
shared set of mental models. This new organizational state was illustrated through numerous 
observations and comments made by both committee members and important stakeholders not directly 
involved with the evaluation exercise. For example, committee members from national organizations 
were faithful to the view that their organizations should be recognized as being the key partner in the 
system, given their legislated connection to program participants. “The role [CNT] should be playing 
in the system is that of providing resources and making interventions like [the evaluation project] to 
assist all the partners facilitate their program. It should not be a top down system where [CNT] drives 
the program from their lofty command center in Ottawa. In fact the preliminary report really should be 
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[CNT’s] strategic plan for future change” (observation field notes. May 30, 1996). Realizing the 
political fallout of making such a significant shift, one member cautions: 

There is going to be a gut reaction at [CNT’s] board level. [The shift] dilutes a little bit of the 
power from [CNT], So who knows what the political response will be to that. I mean, because 
that’s where policy is formed, right? At the board table. (TP 4, steering committee member, Tom) 

For other members of the organization, the experience of working and interacting with many partner 
organizations throughout the evaluation project resulted in similar views being formed. In their words: 

I think the organization is not so centered on itself, it recognizes the importance of key players 
outside the organization and how important it is to get them to contribute to the changes being 
made so that they will then buy in. So [the organization] has a greater understanding of it’s 
role as a service organization. In the past it was "we are here to implement changes and here 
are the changes so you do it..." So that’s a fundamental shift. It’s not 1 00% that way, I just see 
the organization going in that direction. I think this project really helped to give it a big push. 

(TP 4, steering committee member, Brenda) 

It is worthwhile to note how these perceptions transcended the committee to other key 
individuals within CNT as well as the system. The organization’s long-time president confirmed that a 
change in the traditional way of doing business was likely. He too seemed to be caught up in the 
process used to conduct the evaluation. When asked if “the partners would expect more involvement 
in the future,” he replied: 

Yes, I think the trend has been set. There is no question that in the early years [CNT] was not 
only the engine for driving these projects and doing the actual work, [CNT] did a great deal of 
decision making in a fairly authoritarian fashion— it did get things done because it was 
authoritarian. Now we have an expectation by the partners, as a result of [the project], that a 
more democratic process will be used in the future. (TP 4, president of organization) 

The president also provided some intriguing personal reflections on other important effects he noticed. 

In his words: 

...[the project] has unearthed some leadership which was pretty well disguising itself as being 
dormant. Opportunities have been provided for individuals with capabilities to demonstrate 
those capabilities, and that’s what this very time consuming, involvement process has allowed 
to happen. (TP 4, president of organization) 

As the above quotes imply, the support provided by CNT’s president was very important to 
fostering the high levels of learning that appeared to have resulted as a consequent of this project. The 
president, throughout the study, stimulated organization members to seriously question not only the 
purpose and role of the program, but the organization itself. Interview data made mention of the 
president’s personal views regarding the need for CNT to change “its way of doing business.” He took 
it upon himself to encourage others in the organization to recognize and deal with the strategic future 
of the organization including how CNT would be able to meet the expectations of the partner 
organizations (e.g., what skills would staff need to possess and/or acquire). At one point leading up to 
the project’s final report, the president actually chastised some members of the steering committee for 
not having the confidence of recommending fundamental program and/or system changes if they felt 
them to be in order. 
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It is interesting to note how the president came to hold the view that CNT had to undergo a 
significant change to its operating style and/or its culture. Simply, he acquired his new insight from 
sharing information with members at all levels within organization. The president 
fostered this organizational climate through his active interest in the issues being discussed and by 
publicly acknowledging the intellectual abilities of the members and encouraging their work. 

Two other variables are worth noting. First, the decision of the federal government to reduce 
funding to CNT and the partner organizations triggered those partners who were responsible for 
program implementation to re-examine their roles and consider new strategic alliances to ensure the 
survival of existing programs. This period of reflection and examination led to a series of first-ever 
discussions about the role CNT held within the system, especially as it related to the power it 
possessed vis-a-vis the national organizations. As one member from a national organization’s 
reported, “the baseball bat held by the organization all these years is going or is gone.” The notable 
second variable, which is also tied to the funding issue, relates to the establishment of a mentorship 
system among the national organizations to assist one another cope with the inevitable program and 
system changes set to come. With the federal government and CNT losing it’s positional power within 
the system combined with new strategic alliances being formed by the partner organizations, the stage 
would seem to be set for significant organizational change. At the very least, partner organizations 
were now demanding and receiving an opportunity to influence both program and organizational 
decisions. Clearly, a collaborative decision-making process was emerging and taking hold; this 
process was effectively replacing the traditional practice whereby CNT would produce policies that the 
national organizations would be expected to implement with little or no involvement in their 
formulation. 



Discussion 

Cousins and Earl (1992) were among the first to advance the notion that participatory forms of 
evaluations should be re-conceptualized due to their effects not only on the program under study but on 
the organization within which the program is situated. The authors proposed that due to the 
partnership which is developed (between the trained evaluation personnel and practice-based decision 
makers) and nature of the work involved in the conduct of the evaluation, participatory evaluation can 
stimulate the kinds of social interaction and dialogue necessary for improved organizational learning. 

While several other authors (e.g., Lafleur, 1995; Owen & Lambert, 1995; Preskill, 1994) 
support this direction in the literature, clarity as to what ‘organizational learning’ means, which 
individual, and/or group processes stimulates its development and, more importantly, their differential 
effects on learning has received limited attention. For example, what do researchers’ mean when they 
state that organizational learning has been enhanced or reduced? The construct is a dense one and the 
elements associated with organizational learning are somewhat nebulous. As was observed by 
Huberman (1995), organizational learning “is a slippery measure, and I am one of several who is wary 
of it” (p. 103). Consequently, researchers have had difficulty both defining and associating 
organizational learning outcomes with specific elements of the participatory evaluation process. 

The present study has taken a step, albeit tentative, towards addressing this problem by 
describing how one organization increased its capacity to take effective action - to learn. It 
conceptualized organizational learning by delineating or breaking apart the construct into four 
dimensions and four processes. The study also sheds some light on how a participatory evaluation 
intervention in a national, not-for-profit training organization acted to facilitate the transfer of 
individual mental models and beliefs to an organization. If organizations ultimately learn via their 
individual members, then it is appropriate that we first frame our discussion of the study’s results in 
terms of how the participatory process acted to facilitate individual learning. We will then and 
describe how this learning was transferred and integrated into the organization. 




18 



13 



One approach to individual learning that is consistent with the underlying philosophy of 
participatory evaluation has evolved from David Kolb’s work in this area. Kolb’s (1984) experiential 
learning cycle theory advocates that individuals should be required to test out the conclusions that they 
reach through active experimentation. Action is necessary for it serves to both test the interpretations 
made and to generate new information that continues the learning process. Individual learning is 
thought to be dependent on action because: 1 ) a gap exists between stored knowledge and knowledge 
required to act effectively; 2) the organizational contexts are constantly changing; and 3) it is necessary 
for the organization members to codify effective behavior (integrated into existing memory) so that it 
can be reliably repeated (Argyris, 1993). 

The collaborative requirements of this particular evaluation placed specific demands on each 
member to actively engaged in various steps of the intervention - constructing instruments, analyzing 
data, presenting reports, continuously debating fundamental assumptions. These actions, particularly 
the latter two, assisted individuals to alter and/or refine their mental models through a personal process 
of actively integrating viewpoints from many sources and then testing them in a public domain. This 
validation step, as highlighted by the findings, helped members to alter existing frames of 
understanding and move beyond conventional thinking and to challenge current paradigms. 

As argued previously, organizational learning is more sophisticated in terms of the process 
required to effect change than is the case at the individual level. Once new individual mental models 
are established, it is crucial that they be made explicit if new shared/organizational mental models are 
to be formed (Kim, 1993). Dixon (1994) refers to individual mental models that are withheld from 
other members as ‘private meaning structures’. These models are withheld for a variety of reasons that 
extend from respecting the sources of information that was given in confidence to a recognition that 
certain information carries significant political influence/power. Regardless, organizational learning is 
dependent on establishing an environment whereby 1) individuals make their mental models available 
to others, 2) others in the organization are able and motivated to make use of them, and 3) individual 
learning is retained by the organization for future use. 

The efficacy of internal participatory evaluation in facilitating the necessary environment that 
would support these three inter-related activities is still open to debate and likely to depend 
significantly on context. However, evidence was provided in the present study to support the 
conclusion that if conducted in a specific fashion, by the right person, over a long enough period of 
time, process/organizational effects can be observed. Sensitive, micro-political information can be 
unearthed and positioned in the public (organizational) domain that will challenge the individual 
and/or collective view of reality. Key decision makers/stakeholders can be stimulated to reflect 
intently on what these differences are and the significance they hold. The participatory process has 
also been shown to generate a high degree of personal motivation to engage in an ongoing debate (in 
this case approximately two years) with colleagues to resolve differences of opinions and to work 
towards a set of shared assumptions. And finally, that these shared models can then be stored in the 
organization’s memory for future use. 

While it is obvious that one study cannot cast this form of evaluation as a panacea for those 
wishing to enhance an organization’s learning capacity, it does offer hope. The study should also give 
future researchers some guidance as to the power of longitudinal designs assisting in developing 
understanding about the use of evaluation findings and processes in the practical milieu. 

Implications for Research 

The findings of the present study have provided a much more positive assessment of the impact 
of participatory evaluation on organizational learning. Although the study tracked a number of variables 
linked to organizational learning capacity, the conceptual framework developed to guide this study was 
preliminary and should be therefore considered as a catalyst for future research. For example, given the 
important role that the context of the evaluation plays, would the conditions under which organization 
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learning appears to occur in a not-for-profit organization differ from conditions in the private sector? 
What are the factors likely to be associated with higher levels of organizational learning in settings 
where profit is the goal. We also need to know more about practice. What is it that evaluators do to 
facilitate the organizational constructs investigated in the present literature? While the present study 
provides a starting point, consideration could be given to debriefing experienced evaluators who have an 
interest in the organizational consequences of participatory evaluation. 
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EXTERNAL ENVIRONMENT 
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Fi gure 1. Conceptual framework: Internal participatory evaluation as an 
organizational intervention. 
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Figure 2. Make-up of the evaluation project’s steering committee 
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TABLE 1 



Summary of Evaluation Study: Time Periods & Purpose 



Time Period 


Dates 


Description 


TP 1 


< May 10, 1995 


baseline description 


TP 2 


May 1995 to 
October 1995 


finalize design; initiation 
of data collection 


TP 3 


October 1 995 to 
April 1996 


completion of data collection; 
initiation of 
data analysis 


TP 4 


April 1996 to 
October 1996 


completion of data 
analysis; report preparation 


TPS 


October 1996 to 
April 1997 


impact of evaluation findings; 
process use 
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